Exploring the Depths of Recurrent Neural Networks with Stochastic Residual Learning

Recent advancements in feed-forward convolutional neural network architecture have unlocked the ability to effectively use ultra-deep neural networks with hundreds of layers. However, with a couple exceptions, these advancements have mostly been confined to the world of feed-forward convolutional neural networks for image recognition, and NLP tasks requiring recurrent networks have largely been left behind. In this paper, we apply two recent innovations in ultra-deep convolutional networks, ResNets and stochastic depth, to RNNs used for sentiment classification. We also add a new innovation, stochastic timesteps, which is similar to stochastic depth but over horizontal timesteps rather than vertical layers. We achieve classification accuracies on the five class, fine-grained Stanford Sentiment Treebank that are very close to state of the art, without using the parse tree information utilized by current SOTA methods. We believe that these results bode well for the potential of ultra-deep networks in recurrent and NLP settings in addition to their existing uses in feed-forward and computer vision ones.